Imager system with median filter and method thereof

ABSTRACT

An imaging system configured to display a captured image on a display having a lower dynamic range than the imaging system is provided, wherein the imaging system includes a high dynamic range imager configured to capture at least one high dynamic range image, and a processing device communicatively connected to the high dynamic range imager, wherein the processing device is configured to reduce color speckles caused by errant interpolated pixel values and reduce occurrence of errant values of alternate color while maintaining at least one of sharpness detail, edge detail, and generate smoothing color rendering in a displayed image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/530,947 filed on Jun. 22, 2012, now U.S. Pat. No. 9,769,430, whichclaims the benefit of U.S. Provisional Application No. 61/500,418 filedon Jun. 23, 2011, by Jon H. Bechtel et al. and entitled “MEDIAN FILTER,”and U.S. Provisional Application No. 61/544,315 filed on Oct. 7, 2011,by Jon H. Bechtel et al. and entitled “MEDIAN FILTER,” the entiredisclosures of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to a median filter, and moreparticularly to a median filter used in a system that displays an imagecaptured by a high dynamic range imager.

SUMMARY OF THE INVENTION

According to one aspect of the present invention, an imaging systemconfigured to display a captured image on a display having a lowerdynamic range than the imaging system, the imaging system includes ahigh dynamic range imager configured to capture at least one highdynamic range image, and a processing device communicatively connectedto the high dynamic range imager, wherein the processing device isconfigured to reduce color speckles caused by errant interpolated pixelvalues and reduce occurrence of errant values of alternate color whilemaintaining at least one of sharpness detail, edge detail, and generatesmoothing color rendering in a displayed image.

According to another aspect of the present invention, imaging systemconfigured to display a captured image on a display having a lowerdynamic range than the imaging system, the imaging system including ahigh dynamic range imager configured to capture at least one highdynamic range image, and a processing device communicatively connectedto the high dynamic range imager, wherein the processing device isconfigured to find a median from a set of pixel related values andprovide an enhanced pixel metric based at least partially upon modifiedratios of pixel color component to at least one of pixel color componentand luminance values.

According to yet another aspect of the present invention, an imagingsystem configured to display a captured image on a display having alower dynamic range than the imaging system, the imaging system includesa high dynamic range imager configured to capture at least one highdynamic range image, and a processing device communicatively connectedto the high dynamic range imager, wherein the processing device isconfigured to rank each element of an array with respect to an orderingcriterion and select a median element.

According to another aspect of the present invention, an imaging systemconfigured to display a captured image on a display having a lowerdynamic range than the imaging system, the imaging system includes ahigh dynamic range imager configured to capture at least one highdynamic range image, and a processing device communicatively connectedto the high dynamic range imager, wherein the processing device isconfigured to filter successive pixels so pixel related valuesintroduced into neighborhood of pixels over which a median or value ofdesignated rank is chosen are used with related comparison results.

According to yet another aspect of the present invention, an imagingsystem configured to display a captured image on a display having alower dynamic range than the imaging system, the imaging system includesa high dynamic range imager configured to capture at least one highdynamic range image, and a processing device communicatively connectedto the high dynamic range imager, wherein the processing device isconfigured such that elements with equal values are sorted in aprescribed sequence so each element has a rank that is unique within aset.

According to another aspect of the present invention, a non-transitorycomputer readable medium having stored thereon software instructionsthat, when executed by a processor, cause the processor to display acaptured image on a display having a lower dynamic range than theimaging system, by executing the steps including capturing at least onehigh dynamic range image, and reducing color speckles caused by errantinterpolated pixel values and reduce occurrence of errant values ofalternate color while maintaining at least one of sharpness detail, edgedetail, and generate smoothing color rendering in a displayed image.

According to yet another aspect of the present invention, anon-transitory computer readable medium having stored thereon softwareinstructions that, when executed by a processor, cause the processor todisplay a captured image on a display having a lower dynamic range thanthe imaging system, by executing the steps including capturing at leastone high dynamic range image, and finding a median from a set of pixelrelated values and provide an enhanced pixel metric based at leastpartially upon modified ratios of pixel color component to at least oneof pixel color component and luminance values.

According to another aspect of the present invention, a non-transitorycomputer readable medium having stored thereon software instructionsthat, when executed by a processor, cause the processor to display acaptured image on a display having a lower dynamic range than theimaging system, by executing the steps including capturing at least onehigh dynamic range image, and ranking each element of an array withrespect to an ordering criterion and select a median element.

According to yet another aspect of the present invention, anon-transitory computer readable medium having stored thereon softwareinstructions that, when executed by a processor, cause the processor todisplay a captured image on a display having a lower dynamic range thanthe imaging system, by executing the steps including capturing at leastone high dynamic range image, and filtering successive pixels so pixelrelated values introduced into neighborhood of pixels over which amedian or value of designated rank is chosen are used with relatedcomparison results.

According to another aspect of the present invention, a non-transitorycomputer readable medium having stored thereon software instructionsthat, when executed by a processor, cause the processor to display acaptured image on a display having a lower dynamic range than theimaging system, by executing the steps including capturing at least onehigh dynamic range image, and sorting elements with equal values in aprescribed sequence so each element has a rank that is unique within aset.

According to yet another aspect of the present invention, anon-transitory computer readable medium having stored thereon softwareinstructions that, when executed by a processor, cause the processor todisplay a captured image on a display having a lower dynamic range thanthe imaging system, by executing the steps including capturing at leastone high dynamic range image, and using a second derivative termcalculated from one of a red pixel and a blue pixel for interpolation ofgreen pixel.

These and other features, advantages, and objects of the presentinvention will be further understood and appreciated by those skilled inthe art by reference to the following specification, claims, andappended drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become more fully understood from thedetailed description and the accompanying drawings, wherein:

FIG. 1 is a diagram illustrating a median filter, in accordance with oneembodiment of the present invention;

FIG. 2 is a diagram illustrating interpolation patterns and screeningcriteria for conditional inclusion of selected non-green pixel values inthe calculation are detailed;

FIG. 3 is a diagram illustrating interpolation techniques, in accordancewith one embodiment of the present invention;

FIG. 4 is a diagram illustrating a pattern of ten green pixels, inaccordance with one embodiment of the present invention;

FIG. 5 is a diagram illustrating a Bilinear (BL) classification, inaccordance with one embodiment of the present invention;

FIG. 6 is a block diagram of an imager system, in accordance with oneembodiment of the present invention; and

FIG. 7 is a diagram illustrating a relationship of the luminancereferenced median color filter in accordance with one embodiment of thepresent invention to other high dynamic range image processingoperations that are performed on the image.

DETAILED DESCRIPTION

The present illustrated embodiments reside primarily in combinations ofmethod steps and apparatus components related to a median filter.Accordingly, the apparatus components and method steps have beenrepresented, where appropriate, by conventional symbols in the drawings,showing only those specific details that are pertinent to understandingthe embodiments of the present invention so as not to obscure thedisclosure with details that will be readily apparent to those ofordinary skill in the art having the benefit of the description herein.Further, like numerals in the description and drawings represent likeelements.

In this document, relational terms, such as first and second, top andbottom, and the like, are used solely to distinguish one entity oraction from another entity or action, without necessarily requiring orimplying any actual such relationship or order between such entities oractions. The terms “comprises,” “comprising,” or any other variationthereof, are intended to cover a non-exclusive inclusion, such that aprocess, method, article, or apparatus that comprises a list of elementsdoes not include only those elements but may include other elements notexpressly listed or inherent to such process, method, article, orapparatus. An element proceeded by “comprises . . . a” does not, withoutmore constraints, preclude the existence of additional identicalelements in the process, method, article, or apparatus that comprisesthe element.

With imaging devices adapted to render a color image of a scene, eachindividual pixel is normally provided with a color selective filter thathas a color assigned in a prescribed pattern to pixels of the array. ABayer pattern (named after its inventor) with a repeating pattern ofcolor tiles is normally used and is characterized in consisting ofsquare four pixel groups having diagonally positioned differentlycolored filters (normally red pass and blue pass) in opposing corners ofa square array of four pixel color filters and like colored filters(normally green pass filters) in the remaining pair of opposing corners.Variants of the normal Bayer filter pattern as well as other filterpatterns, for example, filter pattern arrays with other colors for thefilters or with two diagonally opposing red filters and individualdiagonally opposing green and blue filters are considered to be withinthe scope of the invention.

Light of the color passed by the color filter placed on each pixel ismeasured at that pixel location and the other color components for thepixel location are normally supplied by an interpolation process. Withhigh dynamic range images, very high contrast edges that occur, forexample, from specular reflections from a vehicle illuminated by directsunlight normally contain some pixel values with large interpolationerrors and these may show up as bright specks in the image that areoften red or blue. Worse still, in video sequences, colors may alternatefrom red to blue appearing much like a distant emergency light. A medianfilter is applied to values calculated for individual pixels preferablyusing equations that indicate a relationship between color components ofindividual pixels in a neighborhood of the target pixel whose value isto be modified. The median value or other value of specified rank in aset of calculated values is located and selected from among the valuescalculated for the pixels in the specified neighborhood of the targetpixel and the selected value is used to calculate a modified value for aspecified color component or specified color components of the targetpixel in the neighborhood. One such filter that serves to smooth colorrendering and to eliminate many of the interpolated values that areseriously in error is described herein. The filter serves to reducenoticeable color speckles due to errant interpolated pixel values and togreatly reduce the occurrence of errant values of alternating color thatare easily confused with emergency flashers while preserving sharpnessand edge detail in the image and generally smoothing color rendering inthe image.

Examples of designs are taught in co-pending United States PatentApplication Publication Numbers US 2010/0195901 A1 and US 2010/0195908A1, both entitled “DIGITAL IMAGE PROCESSING AND SYSTEMS INCORPORATINGTHE SAME,” which are hereby incorporated herein by reference in theirentirety. These exemplary designs have used arrays of compare values asa portion of an edge pattern detection circuit but these prior artdesigns have not applied such an array to establish ranking order tolocate an element that is ranked at the middle or at another specifiedplace in the array of ranked elements. A report from the Army ResearchLaboratory, Adelphi, Md. 20783-1197, entitled, Review of Bayer PatternColor Filter Array (CFA) Demosaicing with New Quality AssessmentAlgorithms by Robert A. Maschal Jr., S. Susan Young, Joe Reynolds, KeithKrapels, Jonathan Fanning, and Ted Corbin ARL-TR-5061, January 2010,which is hereby incorporated herein by reference in its entirety,teaches the use of a median filter based on the median of the differencebetween red pixel color components and the green color component foreach pixel in the neighborhood of a target pixel wherein the redcomponent of the target pixel is modified to differ from its greencomponent by the median value. A similar modification is made to theblue component of the target pixel using the median of the differencebetween blue pixel color components and the corresponding green colorcomponent for each pixel in the neighborhood of the target pixel.

The device as described herein provides an efficient way to find themedian from a set of pixel related values and also provides an improvedpixel metric based on modified ratios of pixel color component values.The ratio calculations are preferably performed using logarithmicrepresentations of the color components.

The present invention includes a device based on a matrix of comparevalues that may be implemented using a field programmable gate array toefficiently determine the ranking of each element of an array inaccordance with an ordering criterion and to then select the medianelement or another element of a specified rank. The ordering criterionpreferably provides a unique discrete ordering of the values in thearray so that even with various equal pixel values a discrete rankingvalue is assigned to every element of the array. It is preferable tobase the discrete ranking of equal pixel values either on their order ofentry into the array and/or on their position within the array ofcompare values.

In the example of FIG. 1, two different medians or values of designatedrank are identified and selected each from its own set of twenty-fiveelements and each is used to filter the respective red and blue pixelvalues of the target pixel in the array. As a preferred option,filtering is not applied to pixels at the boundary of an image where afull set of pixel based values is not available to populate the arrayfrom which a median or value of specified rank is selected. Filtering ispreferably performed on successive pixels in the array so that pixelrelated values introduced into the neighborhood over which the median orvalue of designated rank is chosen are used along with relatedcomparison results for filtering of several successive pixels. One ofthe features of the invention is that compares are preferably organizedso that only the pixel elements in the subset that are newly introducedto the array are compared against other elements in the array. Thiscriterion may be replaced by one stating that at least a portion ofcompare results are obtained once and used in finding the median orvalue of specified rank for more than one pixel in the array.

An additional feature is that elements of equal value are ranked in aprescribed, logically consistent order so that even elements of equalvalue are sorted in a prescribed sequence so that each element in theranking that is established has a rank that is unique in the set. Thishas the major benefit that a middle element or other element of adesired rank is always present and uniquely specified. This is true evenwhen more than one of a set of equal element values might have beenchosen as the median. In this case, selection of any member of the setof equal values that include the median element provide the same result.A second benefit is that for a comparison of A to B if element A is lessthan element B, then element B is greater or equal to element A so thecompare value of B compared to A is the logical complement of thecompare value of A compared to B and for the order dependent rankingthis is the preferred result to assign to a comparison of B to A. Thus,one comparison of A relative to B may be performed without any separatecompares to determine equality and one compare operation provides thenecessary information to provide the value of the compare in thereversed order by simply taking the logical complement of the compareperformed in the non-reversed order. These are subtle points but ones onwhich both the success and the efficiency of the ranking calculationsdepend. If “1” is used to denote that an element is considered greaterthan another element of the set and “0” is used to denote that theelement is not considered to be greater, then the ranking of an elementis preferably denoted as a numerical value ranging from 0 for the lowestrank to N−1 for the highest rank with N elements in the set. This rankfor a given element is equal to the sum of the set of compare values forthe element as it is compared to each of the remaining N−1 elements inthe set.

The median filter is preferably used to modify or filter colorcomponents of the pixel in the middle of a rectangular or square array(the device of this invention is not limited to these configurations.)of pixels with pixels in the array serving as the base for the median orranking calculation that is preferably used to modify one or more colorcomponents of the pixel that is in the middle of the array. In apreferred configuration, the filtering operation is applied successivelyto pixels in a row using the median of a five by five array of pixelrelated values. As one advances from applying the filtering operation toone pixel to application to the next pixel, a new column of five pixelrelated values are calculated and introduced into the array 107 fromwhich the median is chosen and the oldest column of pixel related valuesis discarded from the array. In the example of FIG. 1, values in column0 of array 107 are overwritten by corresponding values shifted fromcolumn 1, as values from column 2 are shifted to column 1, values fromcolumn 3 are shifted to column 2, values from column 4 are shifted tocolumn 3, and newly calculated values from which the median is selectedare shifted into column 4, all in row-wise shift register fashion. Inthis arrangement, each column of pixels serves as part of the array 107from which the median is selected for five successive applications ofthe median filter to five successive pixel values.

As illustrated in FIG. 1, the compares are arranged to reuse bothcompare results and/or partial sums of the compare results (or comparerelated values) as they apply to the set of compare values for eachmember of the array with other members of the array. The final rankingfor each given element of the array is preferably calculated as the sumof the number of true compare results for an ordered comparison of thegiven element to each of the other elements of the array. In a separateoptional improvement, an additive constant is used to reduce the colorcontent of pixels with a low numerical value in order to reduce colornoise in dark areas of the image that are most subject to color noiseartifacts. Where logarithms are used, the addition of a positive nonzerointeger value to non-negative pixel values also serves to preventattempting to take the logarithm of zero. With the preferred log basedcalculation, the values calculated for the median selection are directlyrelated to the logarithm of the ratio of the pixel's red color componentvalue to the pixel's green color component value for the red filteringoperation and to the logarithm of the ratio of the pixel's blue colorcomponent value to the pixel's green color component value for the bluefiltering operation. With high dynamic range images, some of the ratiosmay be very large or very small. In reality, the combination ofelectrical crosstalk, or optical crosstalk, and of limited rejection ofnon-red light by the red pixel filters, non-green light by the greenpixel filters, and non-blue light by the blue pixel filters place apractical limit on the ratio of color component values that an imagerwill render even for large uniform color blocks of a primary color.There may be little benefit in allowing the ratios of the red or bluepixel component values to their green pixel component value to exceedthis device limit by very much. Such a limit will also limit the pixelbit width of numbers needed to convey the color information. In thelogarithmic based color representation the limit on the ratio offiltered color to green or green to filtered color component may beclamped by simply clamping the resulting logarithmic value to staywithin prescribed positive and negative bounds. For example, when usinglog to the base 2, clamping the result to an upper bound of 5 and alower bound of −5 limits the ratio of the color component to green to arange of 1/32 to 32.

In FIG. 1, array 107 indicates 5 rows each with 5 shift register linkedstorage locations arranged to store the array of 25 multi-bit valuesfrom which the pixel related value of the specified rank, preferably themedian, is chosen. A new column of pixel related values is calculatedand input to column 4 (S₀₄, S₁₄, S₂₄, S₃₄, S₄₄) of array 107 whileshifting each of the rows in a row-wise, left shift operation so thatthe oldest elements in column 0 (S₀₀, S₁₀, S₂₀, S₃₀, S₄₀) areoverwritten. Filtering operations are preferably excluded for boardercases where the matrix of values S_(ij) from which median elements areselected is not fully populated. Color components of the pixel relatedto S₂₂ are preferably the ones to which the median calculations areapplied. The triangular shaped compare value array in value rankingcomputation circuit 100 and the supporting array 107 from which themedian or ranked value is selected and calculation and selection circuit136 are preferably provided in duplicate so that median filtering may beprovided for a first color component, preferably red, as indicated in108, and for a second color component, preferably blue, as indicated in109. A number of variants of the median based filter may be usedincluding the one described in the Maschal et al reference. However,especially for high dynamic range images, a ratio-metric function ofpixel component values is preferable to a difference based value. Infield programmable logic circuits, it is often more efficient to computelogarithmic and anti-logarithmic values than to perform direct division.Furthermore, with signed values, high dynamic range liner pixel valuesmay exceed 24 bits and may be more compactly represented as a ratio inlogarithmic form thereby reducing the size of the computational circuit100 making it preferable to use the log form. log 2 is used in FIG. 1 toindicate preferable use of log to the base 2. The invention is notrestricted to use of red, green, and blue color components or even togreen as the reference color component. However, the reference color ispreferably the one most directly related to pixel luminance since beingunfiltered in the median filter operation above, this will help topreserve visible edges. Secondly, it is preferable to use the colorcomponent having the smallest interpolation artifacts as the referencecolor which is again green when using a RGGB Bayer color filter arraywith twice as many green filtered pixels as red filtered or bluefiltered pixels in the array. Prior to calculation of the median valuesit is preferable to perform interpolation to provide a full set of red,green, and blue pixel color components R_(ij), G_(ij), and B_(ij) foreach pixel location in the pixel array. For the red pixel median filterS_(ij) components are calculated preferably using red and green pixelvalues as indicated in 108 for the red pixel median filtering circuitand for the blue pixel median filter S_(ij) components are calculatedpreferably using blue and green pixel values as indicated in 109 for theblue pixel median filtering circuit. RMF22 indicates the median filteredred component to replace the red component of the pixel corresponding toS₂₂ in the red median filter calculation, and BMF22 indicates the medianfiltered blue component to replace the blue component of the pixelcorresponding to S₂₂ in the blue median filter calculation. Note thatvalues calculated for S_(ij) are different for RMF22 than for BMF22.

The circuit 100 preferably operates on pixel based calculated valuesfrom the array 107 and there are preferably essentially duplicateinstantiations of the circuit, one to handle median or other rankingcalculation for one color component as indicated in block 108, and theother to handle similar calculations for another color component asindicated in block 109. A usual way to calculate a median value or avalue of specified rank from a set of values is to sort or partiallysort either the values or pointers to the values and then to select themedian or the value of specified rank from the list. Note here that thedevice in the invention may be applied, still within the scope of theinvention, to obtain the median for an even number of elements in whichcase there is no single middle element, and the median may be calculatedas the average of the value that ranks just below the middle and thevalue that ranks just above the middle. In the preferred configuration,a matrix of compare values or of partial sums of compare values areassembled based on comparisons of new elements entered in the array 107with elements already in the array 107 and partial sums of compareresults characterized in that they are partitioned into subgroups suchthat at least a portion of the subgroups of partial sums remain validfor more than one of the median calculations as portions of the elementsover which the median or specified ranking is performed are replaced bynew values while a portion are retained in successive median or rankingcalculations.

In a specific example depicted in FIG. 1, an array of pixel relatedvalues 107 for which a median and/or other ranking is calculated isdepicted if FIG. 1. The values are organized in five rows and fivecolumns and the median or ranking calculation is preferably performedafter entering a new set of values at column 4 and shifting previouscolumns of values left one column position so that the previous set ofvalues in column 0 is lost. Values previously in columns 1 through 4remain but now with each in the next lower column position so theyoccupy columns 0 through 3. Calculation block 100 provides comparisonindications preferably in the form of binary 1/0 values. Selectioncircuit 110 selects one of the five elements from column 4 as indicatedby inputs 111 and presents the five column values in a sequence to becompared with other elements in the array.

S₀₄ is compared with all 24 elements in the array and S₁₄, S₂₄, S₃₄, andS₄₄ are then successively selected by selection circuit 110 for compareand are compared, respectively, with 23, 22, 21, and 20 other elementsin the array. The array is triangular because, as indicated above, thecompares are preferably structured to logically support a specifiedordering so that equals will still be assigned a discrete rankingdepending on their position in the array of compare values and/or theirorder of entry into the array and support for the ordering is providedby using the logical complement of the compare indication when the orderof comparison is reversed. For example, each of the 24 compare circuitswith compare circuit 122 shown on the right may be configured toindicate that the element selected for comparison, 132, is greater thanthe element against which it is compared by output of a compare value of1 for the greater than compare condition and an output of a comparevalue of 0 otherwise when the element selected for comparison is lessthan or equal to the indicated element against which it is compared.Then the value of the compare for a reversal in the order of thecomparison is the logical complement of the compare value indicated bythe circuit. Since compare values specified in the reverse order are thelogical complement of compare results obtained in the original order,the triangular matrix contains all of the necessary compare results toobtain indications the result of the compare of any value in the arrayto any other value in the array. The compare indications calculated inthis sequence of compare operations provide all of the compare valuesneeded to rank the elements of array 107.

Values of all elements of the array except S₀₄ are inputs to individualcompare circuits that are grouped by their column position in array 107and input to the comparators at 112, 113, 114, 115, and 116. For each ofthe subgroups for a particular column, inputs are grouped by rowposition in 107. The row and column designation from array 107 ofindividual input compare values against which the selected compare value132 is compared are also used as column designations for the matrix. Therow designations of S₀₄ through S₄₀ are also grouped by their columndesignation in array 107 in the same order as in array 100 going fromtop to bottom as the column designations in going from right to left sothe row column locations representing the self compares of elementsagainst themselves occur at the right ends of the rows of the triangularmatrix. Element S₄₀ is included after S₃₀ because the sum ofcomplemented compare indications in column S₄₀ indicates the rank ofelement S₄₀ as will become clear from the ensuing description. The rowsrepresenting compare results for elements from columns 4 through 0 ofthe array 107 are grouped as indicated by brackets 117, 118, 119, 120,and 121. The row and column designations indicate respectively the valuebeing compared for the row and the value against which it is comparedfor the column. The larger black dots like 125 indicate compare valuesthat are computed and open dots like 127 represent the matrix locationof the self compare of an element to itself. With the indicated greaterthan compare criteria, these would be 0 and are only place holders whosevalues do not need to be used. They are used as row to column transitionplaceholders as will be indicated in the description below.

The complete list of compare indications for any element includes all ofthe compare indications in the row for the designated element up to theself compare indication that is preferably not included and continuewith the logical complements of compare indications from rows above thisrow that are in the column with the same element designation. The selfcompare indication is at the intersection of this row/column pair thatis associated with a particular element. It is preferable to partitioncompare results into groups that are related to sets of pixel relatedvalues that are entered and/or moved within the array betweencalculations of a median or ranking value for successive pixelsrepresented by the array of calculated values.

In the array 100, there are five triangular sub arrays of elements 102,103, 104, 105, and 106, each representing compares of elements withother elements in the same column and including the place holder for aself compare that is preferably not performed. The five “L” shaped boxessimilar to 129 in sub array 102 contain the compare indicationsassociated with a particular element of the array 107. The middle “L”shaped box at 129 contains the compare indications in the horizontalportion of row S₂₄ for which S₂₄ is compared with other elements in thecolumn, the self compare place holder, and elements from column S₂₄ forwhich other elements from other rows are compared against S₂₄. Thesummation of four compare related values enumerating the comparisons ofS₂₄ with other elements in the same column is obtained by summinglogically true occurrences of the compare indications in the row portionof the group with the sum of logically true occurrences of the logicalcompliments of the compare indications in the column portion ofcomparison where the original compare is for that of other elements inthe column against S₂₄. In a preferred implementation, the sum oflogically true indications made up of the sum of compare indications forthe row portion and sums of the logical complements of compareindications in the column portion of the compare indication matrix isregistered in three bit memory 128. Similar sums following similar rulesare made and registered in corresponding memory elements for comparisonsthe remaining four elements S₀₄, S₁₄, S₃₄, and S₄₄ with other elementsof the same column. It should be understood that the geometry of thelayout in FIG. 1 is merely for convenience in description as areparticular numbering designations and the invention is not limited byeither. As will be indicated hereinafter, the rank of each element ispreferably taken as a count or sum of the number of other elements inthe array 100 that are exceeded by the element specified in the row andcolumn pair according to the chosen compare criteria. Here the comparecriteria may include comparison ordering and/or position in the array ofcompare values and the compare criteria or rule is preferably used toestablish a discrete order even when there are one or more subgroups ofelements of equal value in the array 107. The logical complementingdescribed herein is used to transform from a compare result for acomparison performed in the reverse order to a compare result for acomparison performed in the desired order. As the array 100 is shown andfor the ordering preference used, compare indications in rows are in theproper order and are not complemented in the partial compare resulttallies and compare indications in columns are in the reverse order andare logically complemented in the partial compare result tallies. Sumsof designated subgroups of the partial compare result tallies that areindicated are sufficient to calculate the rank of each element soindividual compare indications need to be registered only until they areentered into necessary horizontal and vertical sums and do not need tobe registered at all if incorporation in the sums does not requireregistering of the value. For the greater than criterion used in theexample for the direct compare operation for new elements as they areentered, older elements with row designations that are further towardthe bottom in the list of row designations in the triangular array ofcalculation unit 100 are given higher ranking than equal elements thatare higher in the list of row designations.

There are ten sub arrays of 25 elements four related to Col 4, three toCol 3, two to Col 2, one to Col 1 and none to Col 0. The sum oflogically true indications for the compare of element S₂₄ with the fiveelements of Col 3 is registered in 3 bit register 123 and the row of thefive compare indications used for the sum is contained in the outlinedhorizontally positioned box 124. The sum of logically true indicationsof the logical complements of compares of the five elements in Col 3against element S₂₄ is registered in 3 bit register 131 and the columnof five compare indications is contained in the outlined verticallypositioned box 130. The four remaining vertical sums and the fourremaining horizontal sums for 101 are calculated and registered in thesame manner as are partial sums of compare indications for the remainingthree sub arrays of elements for the five rows of compare indicationsrelated to elements in Col 4 (117). Select circuit 110 is used in thepreferred embodiment to cycle through the five elements of Col 4 ofarray 107 to obtain the 110 compare indications and their logicalcomplements for compares in the reverse order. Note that each comparevalue is used twice, once for a comparison in the order used in thecomparison and once in complemented form for a comparison in reverseorder. As will be explained below, the compare indications or preferablythe sums of compare indications provide the data for the 190 additionalcomparisons associated with columns 3, 2, 1, and 0 of array 107.

With the organization of compare results indicated in FIG. 1, when thecalculation of median values are completed for a pixel in the row, ifthere are additional pixels in the row, all columns of elements in array107 are shifted left overwriting values for Col 0 and entering newvalues for Col 4. Inspection of the triangular array 100 with associatedcolumn designations for element designations for both the columns (116,115, 114, 113, and 112) and rows (117, 118, 119, 120, and 121) of thearray 100 reveals that the column shift of columns in 107 correspond todiagonal group shifts one column position down and one column positionto the left in the array 100. This indicates that compare values orpreferably the partial horizontal and vertical sums of compareindications remain valid when shifted down and to the left by one columnposition and by one row position dropping values that would be shiftedout of the array 100 and calculating compare results and partial compareresult sums for only the first five rows of the array 100 that arevacated by the diagonal shift operation. Stated in a more generalcontext, the positional designations of compare values from previousoperations that correspond to elements that are retained for a newranking operation are changed to reflect the new position of theelements on which the compare values are based in the array 107.

Block 136 lists the final operations to calculate a rank for eachelement of the array 107, to indicate the element/elements of thespecified rank/ranks, and to select it/them as needed for specifiedoperations. From array 100, the rank for each element listed in the rowand column headings is the sum all of the partial sum terms that appeareither in the row or the column for the specified term. In the example,there are five partial sum terms for each element. The circuit ispreferably configured to compute the sum for each of the twenty fiveelements and to test the value of each sum for equality with thedesignated value or values to flag the one or ones having the designatedvalue or values. The preferred circuit is configured so that everyindication of rank is discrete, that is, every value from 0 to 24 occursexactly once as an indication of the rank for each calculation of theset of rank values. In the preferred implementation a select circuit ispreferably provided to select the median value having rank 12 and supplyit as the value to use for median (S_(ij)) in the equation for RMF22 forcalculation of the red element filtered value in block 108 or for BMF22for calculation of the blue element filtered value in block 109.

Several options that may not be directly linked to the median or rankselection based filter above are described in what follows.

The median based filter described above is effective in removing colorfrom isolated pixels but the luminance of the pixel is set primarily bythe value of the green pixel that is not changed in the median filteringof the red and blue pixel components. With distracting color artifactsremoved, very bright, isolated pixels from specular reflections ofsunlight are still somewhat distracting. As an option, after using greenpixel component values to calculate the median and preferably with greencolor components in the logarithmic form, it is beneficial to filter thegreen components by locating the isolated bright green pixel colorcomponents and reducing their brightness preferably using the originalgreen component for the calculation of S_(ij) in the equations in blocks108 and 109 of FIG. 1 and then using the value of reduced magnitude forG₂₂ in calculation of RMF22 in 108 and BMF22 in 109. In this way, coloris effectively scaled while reducing brightness of the isolated spots.As an optional but beneficial addition to median filtering of the redand green color components of pixel values, green color components maybe scanned to locate localized peaks and the value of these peaks may beclamped or otherwise limited in brightness level relative to neighboringpixels to subdue distracting bright peaks. Since the most distractingbright spots come from specular reflection of direct sunlight on a clearday and with the sun as a reference, such spots are very bright. Thus,as an option, reductions in pixel brightness may be limited to pixelswhose level exceeds a threshold value or other brightness criteria.

In one implementation, preferably with green pixel components inlogarithmic form, G₂₂ is first sampled for use to calculate the separatevalues of S_(ij) for the filtering of the red pixel color component forthe pixel corresponding to location S₂₂ in block 108 of FIG. 1 and forthe filtering of the blue pixel color component for the pixelcorresponding to location S₂₂ in block 109 of FIG. 1. Then the max valueof the four or optionally eight nearest neighbors of G₂₂ may becalculated and the value of log 2(G₂₂) may be clamped to this value plusa constant z (as an example, z might be set to 2) and used to replacethe green component of the pixel corresponding to S₂₂ and also tocalculate RMF22 and BMF22 so that the filter red and blue colorcomponents 108 and 109 of FIG. 1 are adjusted to generally preserve thecolor ratios of red to green and blue to green when the value of G₂₂ isadjusted. In the antilog form this will clamp the value of the greenpixel G₂₂ to be no more than 2^(z) times greater than the maximum valueof the green pixels included in the set of neighboring pixels from whichthe maximum pixel value is selected. With z equal to 2, this would be alimit of 4 times greater. Additionally, for the implementation justdescribed, the relative values of the red, green, and blue colorcomponents are preserved with the limiting of the green pixel componentso that the effect on its color is minimal.

In a second implementation, to reduce the effects of pixel reading noisein low light level during the interpolation stage to supply missingcolor components, for pixel locations where the reading value is low(below 32 for example) the interpolation system reverts to aninterpolation mode that provides interpolated values that contain lowerand/or less noticeable noise levels. In one implementation, for pixelswhose readings are below 32, bilinear interpolation is substituted for amore complex interpolation algorithm that may include edge detectionand/or inclusion of pixel values for color components that are differentfrom the one being interpolated. The reasons are that edge detectalgorithms may interpret noise as edge patterns and create edge likeinterpolation artifacts and the coefficients of pixel values that arenot of the color being interpolated are frequently based on estimates offirst or second spacial derivatives of color components and thesedigitally sampled derivative values amplify noise. The bilinearinterpolation based on averages of nearest pixel values using pixelshaving filters with the color being interpolated use smoothing averagingeffects and do not add to noise levels through use of color componentsof colors that differ from the one being interpolated and do not useedge detection each of which add noise as indicated above.

In prior art, some outside mirrors are tilted downward when the vehicleis placed in reverse gear and the images from some cameras specificallyapplied for use when the vehicle is in reverse are known to provideadaptive fields of view that are dependent on vehicle speed anddirection of travel. In a new device, a camera and display are providedto replace an inside, rearview mirror and like the mirror that theyreplace, they have a relatively large width to height aspect ratio. Whenthe vehicle is placed in reverse gear the mirror is configured todigitally select an image that is included in the camera's field of viewbut that is different from the one used for normal forward travel of thevehicle. This image includes a view of the ground or roadway that iscloser to the vehicle to assist the driver in backing. The selection mayoptionally be varied over time or using factors such as vehicle speed orinput from proximity detectors that might pan or switch to a field ofview likely to include a view of the detected object.

One of the most objectionable consequences of saturating parts of animage is that color is normally lost or severely distorted. With imagershaving very high dynamic range there are often very bright areaspossibly including direct view of the sun or direct view of other lightssuch as headlights, signal lights, stop lights, emergency flashers, orstreetlights that are usually far brighter than the rest of the scene.For use in vehicles, it is particularly problematic to turn a color towhite for full saturation or to some errant color for partial saturationwhen the original color identified light from a stop light, turn signal,emergency flasher, taillight, or other light that is normally identifiedat least in part by its color. Unlike normal cameras that do notnormally capture color information in saturated portions of a scene,imagers having a very high dynamic range capture color information muchof the time for many of the lights referred to above. In a preferredversion of the present device, rather than clipping individual colorcomponents that exceed the output pixel value range, it is preferable tolimit values in a way that applies the adjustment to all of the colorcomponents of a pixel having a color component that exceeds the outputrange preserving the color of the pixel. Then operations such as tonemapping used to map the dynamic range of the captured image to thedynamic range of the output space may be used with somewhat lowercompression settings without losing color information from lights thatare frequently far brighter than anything else in the scene.

In a preferred implementation of color preserving pixel value limiting,pixel color components are preferably encoded in logarithmic formpreferably with the antilogs of these values being in a linear orperceptually linear color space. A linear form of RGB is preferred withred, green and blue color components all having values ranging from zero(or one) to the same maximum color component value for the pixel. Foreach pixel the maximum value, in logarithmic form, of the colorcomponents for the pixel is preferably chosen and compared with thelogarithm of the maximum color component value for the output pixels. Ifthe maximum is exceeded, the difference between the logarithm of themaximum pixel component and the logarithm of the maximum output pixelvalue is subtracted from each of the logarithmically encoded pixel colorcomponents. At some point, preferably as part of a lookup table basedconversion to the color space of the output pixel value, thelogarithmically encoded values are preferably converted toanti-logarithmic form or to another desired form. In the implementationjust indicated, each pixel having a value leading to saturation has itscolor components scaled by a common ratio so the maximum color componentfalls just short of saturation. In a modified version of the hardlimiting applied above, the brightest portion of the output value rangemay be reserved for pixel values to which limits are applied and a softlimiting may be applied to scale the brightness of the pixels to whichlimiting is applied so that those that just exceed the lower limitingthreshold are mapped to the lower part of the range reserved forlimiting and very bright pixel values that greatly exceed the limitingvalue are mapped to the upper part of the range.

Another issue is that at night, with dark surroundings, the imagerexhibits its greatest noise and this noise is most noticeable in thedark surroundings making it highly objectionable. It has been found tobe beneficial to subtract a preferably constant value from pixelreadings, preferably prior to color interpolation to hide the worst ofthe noise in the black level so the display will go to its blackestlevel when image illumination is too low to render a good image. Thesubtraction is preferably performed so that the pixel value is notallowed to go below zero and with issues in taking the log of zero ordivision by zero, it is often preferable to set the lowest valueassigned to a pixel after the subtraction to one. The additive constantk used in blocks 108 and 109 of FIG. 1 may be increased to compensate inpart for the effects on color at low light levels due to lowering ofvalues of individual color components in raising the black pointslightly above the level at which noise predominates.

As described in prior art such as Application note KAC-9648 Rev 2published by Kodak in November 2004, the characteristics of filter andimager response to color do not completely match human vision soimagers, like almost all other devices that handle color images,normally benefit from color correction. This was accomplished byconverting colors using a three by three matrix multiplication asoutlined in the application note. Improvement in color was significantwith further improvement achieved by adjusting matrix coefficients tocalibrate them to the specific imaging device.

In FIGS. 2 and 3, a beneficial option to use second derivative termscalculated from red or blue pixel values to improve interpolation ofgreen pixel values is detailed. The option includes a test of themagnitudes of red or blue pixel values that are candidates for use inthe derivative term for the calculation relative to green pixel valuesselected for use in the calculation of the interpolated green pixelvalue. When the magnitude of one or more of the red or blue pixel valuesthat are candidates for use in the calculation of the green pixel valuesare unexpectedly large based on a screening criteria wherein the red orblue candidate values are compared to the relative magnitudes of greenpixel values selected for use in the interpolation, the secondderivative terms are omitted from the calculation. Otherwise the secondderivative term is included. The inclusion of the derivative termimproves the sharpness and resolution of the image but without thescreening criteria their inclusion contributed to substantial increasesin the number of pixels that stood out visually as serious interpolationerrors in very high contrast regions of the high dynamic range image.With the inclusion of the screening criteria, most of the benefit isrealized without the large increase in serious interpolation errors. Thescreening criteria is preferably based on a comparison of the relativemagnitudes of pixel values, preferably of pixel values that arecandidates for use in calculation of the interpolated value rather thanon comparison with predetermined thresholds. Nonzero pixel values in thesame frame may range over more than a million to one in some scenes anda high dynamic range camera for which these calculations were usedprovided nonzero pixel values that ranged over more than 10 million toone in the same high dynamic range image so that predeterminedthresholds applicable to one portion of the scene for purposes of thisscreening would not be appropriate for other very high contrast areasrecorded at much different illumination levels in the same scene.

Referring to FIG. 2, where interpolation patterns and screening criteriafor conditional inclusion of selected non-green pixel values in thecalculation are detailed. With a conventional RGGB Bayer color filterpattern, a pixel C₂₂ for which an interpolated green pixel value iscalculated will have either a red or a blue color filter and pixels C₀₂,C₂₀, C₂₄, and C₄₂ all have filters of the same color as C₂₂. The pixelsC₀₂, C₂₀, C₂₄, and C₄₂ are each two pixel positions away frominterpolation position C₂₂ and may have values that differ radicallyfrom that of C₂₂ in high contrast portions of the image and such radicaldifferences (ratios) often lead to interpolation errors that stand outwhen these values are used in an interpolation equation. On the otherhand, in most other parts of the image, the inclusion of these values inthe interpolation equation leads to more accurate interpolation resultsthat provide significant improvement in both perceived and realresolution of the interpolated image. In FIG. 2, a prior art edgedetection algorithm is used to classify the interpolation asnon-directional or as a diagonal edge in which cases the pattern 200 isused and patterns 201 and 202 are used, respectively, when vertical orhorizontal edge patterns are detected. The cross interpolation is usedwhen the red or blue filtered “C_(xx)” pixels meet the screeningcriteria and the cross (alt) patterns from which the color pixel secondderivative based terms are omitted are used when the “C_(xx)” pixelvalues do not meet the screening criteria. The screening criteria usedin the example are based on comparisons of the relative magnitudes ofgreen and non-green pixels that are candidates for inclusion in theinterpolation equation. In the equations K represents a multiplyingvalue and, as an example, 2 has served well as the value for K in trialsof the screening criteria. 2 is convenient since the pixel values may bemultiplied by 2 by simply shifting them one bit position to the left.The criterion used is to include the color term when none of thenon-green filtered pixels other than the one at the interpolation sitethat are candidates for use in the interpretation calculation are morethan K times larger than the closest green pixel that is used in theinterpolation equation. The conditional statements and equations for thethree pattern specific versions of the interpolation equations at 200,201, and 202 detail the operations.

Much of the detail in FIGS. 3, 4, and 5 is described in priorapplications referenced above, and will not be repeated here. FIG. 3 isdifferent than the prior art for several reasons, including, but notlimited to, first, the cross interpolation equations used to calculateinterpolated green pixel values have been replaced by the conditionallyselected cross and cross (Alt) interpolations described in detail in thedescription of FIG. 2. Second, four categories of cluster patterns arenot included but may be used as an option. Third, coefficients of greenfiltered pixel values used in a number of the interpolation calculationsfor red and blue interpolated values have been scaled to lessen theireffect after it was discovered that this generally reduced the intensityof the Moiré patterns in the interpolated images. Fourth, an additionalBilinear (BL) interpolation category is added.

In FIG. 5, lines 501 through 505 illustrate the Bilinear (BL)classification in item 501. The horizontal edge H and vertical edge Vindications at 516 and 517 have been modified to specifically excludecases where BL is asserted and bilinear interpolation is used or wherethe multiple close spaced edge indications ME or CME are asserted inwhich case the horizontal or vertical indications may be assertedthrough further processing detailed in prior filing referenced herein,and in this implementation are selected as indicated in the prior filingbut are separately labeled as Used for the entire image, bilinearinterpolation has many drawbacks. However, it has an averaging effectthat tends to reduce the effects of noise and its interpolationalgorithms contains no derivative terms that amplify noise.

Furthermore, edge detection algorithms may classify some of the noisepatterns as edges or may detect and thereby accentuate actual edges inimager artifacts. Artifacts introduced by switching integration rangesbecome most noticeable in very low contrast areas of the image such asin views of clear blue sky. Additionally, with an imager for which thesealgorithms are used, there is an indication that the exposure of a pixelis not within its expected range. The out of range indication is mostoften asserted because the light level projected onto a pixel changesduring image acquisition due to being the image of a flashing or dutycycled light or due to being the image of a moving high contrast edge ina scene.

In all of the situations noted above, the bilinear interpolation tendsto soften the interpolation and image acquisition artifacts. The verylow and very high pixel values are detected by direct comparison of thepixel value with K2 and with K3 and the unexpected range indication iscommunicated from the camera as part of the pixel data. Optionally, thisinformation may be obtained by analysis of the raw pixel data todetermine if the pixel reading is in or out of the expected range for aparticular integration period. Features to provide this information inpixel data are provided by only a few devices such as the one describedin references referred to herein. Detection of low contrast regions inthe image is the most difficult and there is synergy in using compareindications used in the preferred pattern based edge detection algorithmwhose operation is outlined in FIGS. 4 and 5 and applied in part in themedian filter depicted in FIG. 1. The median filter in FIG. 1 utilizesthe triangular matrix of compare related values to rank the elements inorder to enable and facilitate selection of an element of specifiedrank. This same mechanism may be added here to utilize the existingarray of compare indications for the green pixels in the pattern arrayto identify the indices i(min) and i(max) of the respective minimum andthe maximum green pixel values in the array of green pixel values. Toadd this capability, the compare indications corresponding to KEY i asindicated in 507 are checked according to the expression 504 to find thevalue of i=i(min) for which all of the compare indications Bij for whichj is not equal to i are zero indicating that Pi is less than (orpossibly equal to) Pj. This check is repeated either sequentially and/orin parallel for KEY i for each value of i to find the index i(min) ofthe pixel with the minimum value in the array (or at least a minimumwhen more than one member has the minimum value) and a selection circuitis provided to use i(min) to select the minimum pixel value Pi(min).Likewise, the compare indications corresponding to KEY i as indicated in507 are checked according to the expression 505 to find the value ofi=i(max) for which all of the compare indications Bij for which j is notequal to i are one indicating that Pi is greater than (or possibly equalto) Pj. This check is repeated either sequentially and/or in parallelfor KEY i for each value of i to find the index i(max) of the pixel withthe maximum value in the array (or at least a maximum when more than onemember has the maximum value) and a selection circuit is provided to usei(max) to select the maximum pixel value Pi(max). Since the compareindications are only one bit each, this logic even when implemented inparallel requires far fewer resources when using the shared array ofpixel compare indications as indicated in FIG. 4 than providing anindependent implementation. In the expression for BL of the example, lowcontrast is indicated by what is effectively a ratio metric comparisonwhere the minimum of the representative pixel values is greater than aconstant K1 times the maximum of the representative pixel values. K1should be less than one and in experiments, 0.75 performed well as thevalue for K1 in experiments. This expression may be rearranged toindicate that the contrast is low when the ratio of the minimum pixelvalue to the maximum pixel value in the set of pixel representative ofthe local contrast level is greater than K1 where K1 is less than one.This is logically equivalent to the form expressed for BL in 501 andmore intuitive but harder to implement.

Equation 502 indicates the value of the compare indication thatindicated that Pi is greater than Pj. This information needs to be knownto select the minimum and maximum values. With the pattern arrays,pattern images that were negatives of other pattern images were oftentreated as equivalents so knowledge of the sense of the compare was notalways needed. This is not intended to imply that a 1 or 0 must beassigned to a particular compare condition but that the value assignedto a particular compare condition needs to be known and appropriatelyused in finding pixels with the maximum and minimum numerical values.

The equation for BL 519 asserts BL for the low contrast or the very lowpixel value or the very high pixel value or the unexpected rangeindication for the pixel value to select the interpolation pattern,preferably bilinear, that produces generally less objectionableinterpolation artifacts in these situations.

In the above, the usual Bayer pattern having two green filtered pixelsfor each red filtered and each blue filtered pixel is assumed forconvenience in description. Further, the bilinear interpolation is usedas the interpolation method to produce fewer artifacts under theconditions detected by expression 501 for BL. The invention is notintended to be limited by these choices. Color filter arrays with otherpatterns may be selected. For example, if a filter pattern similar tothe Bayer pattern but with two red filtered pixels to each green andeach blue filtered pixel is used, the references to green may bereplaced by references to red and the references to red may be replacedby references to green in most of the foregoing description. Statedconditions may be omitted or others added in the expression for BLand/or other interpolation patterns that generally reduce interpolationartifacts for pixels for which BL is asserted may be used. Note that theBL classification preferably applies to selection of the interpolationpattern for two adjacent pixel locations one with a green and the otherwith a non-green filtered pixel.

In conditional selection of the green interpolation pattern, thederivative term is referenced. This is correct for the preferredinterpolation pattern. However not all interpolation patterns usederivative based terms to enhance the quality of the interpolation andthe use of candidate pixels to include in the interpolation forconditional use of non-derivate terms for enhancement of theinterpolation are also considered to be part of this invention.Additionally, as an alternative to comparison of red or blue candidatepixels to green pixels, they may be compared to the central like coloredred or blue pixel or minimum and maximum pixel values in the candidategroup may be compared as alternate screening criteria as a part of thisinvention.

As with most digitally processed images, a sharpening operation toincrease perceived resolution is beneficial. A preferred embodiment isbased on a prior art unsharp mask enhanced by use of an image edge andimage feature classification based mask generated primarily with dataalready provided by the multi-pattern interpolation calculation circuit.The mask is used to restrict application of sharpening primarily tosignificant edges and preferably to further restrict sharpening toexclude areas of the scene known to frequently contain noise or otherartifacts that should not be made more prominent by sharpening. The maskis buffered for synchronization between the interpolation and sharpeningoperations and is used to selectively restrict application of theunsharp mask to areas where edges are detected preferably usingdetection circuits and data also used in the interpolation operation. Asan optional but preferred enhancement, sharpening is further restrictedto exclude areas of the image where sharpening is likely to increasenoise or increase the prominence of other artifacts. Such areas mayinclude low contrast areas, rapidly moving high contrast edges, areasthat image flashing or duty cycled lights, areas with low pixelillumination, and/or areas with extremely bright pixel illumination. Themulti-pattern edge detection described in this specification in relationto FIGS. 3, 4, and 5 already provides indications of the image featuresjust described. In the preferred implementation BL (501 in FIG. 5) isasserted for low contrast, low pixel values, unexpected pixel values fora chosen pixel integration period that are often due to moving highcontrast edges or to flashing or duty cycled light sources. Horizontal Hdesignation 516 and vertical V designation 517 in FIG. 5 explicitlyexclude pixels having the BL designation 501 or the multiple closespaced line designations ME 514 or CME 515. These latter two patternsindicate areas where close spaced edge features make it difficult todistinguish between horizontal or vertical edges and are prone to Moirépattern artifacts that become more prominent when sharpened. Thus, it ispreferable to include pixel locations where ME or CME are asserted inthe classification of pixels for which the sharpening strength isreduced or sharpening is inhibited.

FIG. 6 depicts a system, preferably implemented using a fieldprogrammable gate array or other digital logic based device, thatincorporates shared use of data that indicates the location of edges inan image and/or the location of special image features such as lowcontrast, very low or high pixel values, data indicative of moving edgesor flashing or duty cycled lights, and/or regions of the image likely toexhibit Moiré patterns whose identification is beneficial to the choiceof color interpolation algorithms and also to inclusion, exclusion ormodulation of the strength or type of sharpening or smoothing algorithmsapplied to specific regions of an image. In a preferred embodiment, datafrom a high dynamic range camera 601 is communicated to a preferablypattern based edge and feature detection unit 602. Pixel data ispreferably communicated in a serialized raster format and majorcomponents of the image processing are preferably arranged to operate onpixels from a relatively small region such as a 5×5 kernel to performtheir designated operation.

Additionally, image processing components are preferably arranged in apipelined configuration to perform successive operations with minimaldelay once data for a designated operation is available from the cameraor from a prior processing operation. This arrangement serves togenerally minimize memory requirements for buffering of data and thelatency between the acquisition and the display or other use of theprocessed image data.

In FIG. 6, identification of edges and pixel related image features 602is preferably performed at nearly the same time as interpolation toprovide missing pixel color information in 603. Pixel location relatededge and feature detection data is also communicated from edge andfeature detection block 602 to block 605 in which the data is arrangedpreferably as an image related mask to control the degree of sharpeningapplied to related pixels in the image. It is preferable to arrange abit mask with bits corresponding to individual pixels of the image anduse the bits as indicators to apply or inhibit sharpening at eachcorresponding pixel location according to the value of the bit mask. Inmemory buffer 607, the data is normally buffered just long enough toallow time for corresponding pixels to be subjected to interpolation 603and tone mapping and other image processing 604. Ordering of blocks 606to construct a sharpening edge inclusion and selected feature and 607 tobuffer the data for use in sharpening is optional and is preferablyarranged to minimize buffering memory requirements. Synchronized pixeldata from image tone mapping and processing block 604 and sharpeningmask processing and buffering block 605 are communicated to sharpeningfilter block 608 where the mask or alternately encoded data is utilizedto modulate the degree of sharpening and/or smoothing and optionallychoice of sharpening and/or smoothing algorithms. Such choices betweensharpening and smoothing might follow the pattern of choices between thestandard edge based algorithms and the bi-linear interpolation chosenfor pixels where BL is asserted in FIGS. 3, 4, and 5. Here, for example,the edge inclusion and selected feature rejection mask may optionally bestructured to explicitly indicate BL or other selected featureindications that then may be used both to inhibit sharpening and toenable filtering or smoothing routines.

After optional further processing, image data may be displayed and/orused for other control purposes at 609. Block 430 of FIG. 4 indicates apattern of 10 green pixels used in the preferred multi-pattern edgedetection for image missing pixel interpolation. Pixel interpolatedcolor component values are calculated sequentially for image pixellocations corresponding to locations 460, 461, 462, and 463. The 5 pixelhigh by 4 pixel wide pattern image area is used for green filtered pixellocation 461 and for interpolation at non-green filtered pixel location462 and new patterns are detected at every other pixel location. Theedge patterns detected for array depicted in 430 are used first tosupply edge or other pattern information for the green filtered pixel G6at 431 and then for non-green pixel location 462. Patterns detected andused for each site may be different. In FIG. 3 the diagonal down right,DR, and diagonal up right, UR, indications are only applied for thenon-green filtered pixel location at 462 and not for green filteredpixel location 461. The same horizontal line features in the image maybe detected over as many as 4 consecutive rows of pixels and the samevertical line features in the image may be detected over two consecutiveoperations of the edge detection circuit so when applied to both G6pixel location and the next pixel location 462 for two consecutiveoperations that are two pixels apart, 4 consecutive pixels, (green,non-green, green, non-green), may be flagged as being in the vicinity ofan edge. This is good since sharpening usually accentuates thebrightness level on each side of the edge making the brighter side evenbrighter and the dimmer side even dimmer. Thus, a mask to allow pixelsharpening should span several pixels and extend at least one pixellocation on either side of the edge.

Green pixel location G6 461 is toward the left side of the pattern so itwas found to be preferable to also flag the next pixel location to theleft (460) to enable application of sharpening at the pixel locationcorresponding to 460 for an edge detected in association with 461.Similarly, non-green pixel location 462 is toward the right side of thepattern so it was found to be preferable to also flag the next pixellocation to the right (463) to enable application of sharpening at thepixel location corresponding to 463 for an edge detected in associationwith 462. For the example, lens sharpness was good so a Gaussian blurradius of one pixel or less was used for the unsharp mask. When largerblurring radii are used, it may be advantageous to increase the pixelsflagged to enable sharpening in the neighborhood of a pixel for which anedge is detected and this neighborhood may extend in the vertical aswell as the horizontal direction. In trials, little difference was seenon the sharpening effect on edges in the image where sharpening wasdesired and accentuation of noise in the image and accentuation of otherimage artifacts was significantly reduced by generally restrictingapplication of sharpening to detected edges and further suppressingsharpening in areas where noise and various other artifacts would beaccentuated by sharpening. It appeared that essentially the same imageedge and image feature information served very well for bothapplications saving considerable resources in comparison to separatelygenerating the information for interpolation and for sharpening. Asindicated above, the same information may be used for image smoothing orfor a combination of image smoothing and image sharpening.

A color plane of an image can be transformed by transforming eachindividual pixel value in the color plane by calculating the logarithmof the ratio of the individual pixel color component value to itsrespective luminance or luminance related value. A median filter can beapplied to pixel values in the transformed color plane by replacingindividual transformed pixel values with the median value of transformedpixel values of pixels in a neighborhood of the pixel being transformed.The neighborhood can be approximately centered on and can include thevalue of the pixel being filtered. This will be referred to as aluminance referenced median filter. In one embodiment, red and bluecolor planes are transformed and median filtered as indicated, and greenpixel values are calculated from the pixel's luminance (Y) value and themedian filtered red and blue values or G′/Y is derived from R′/Y andB′/Y. The results of tone mapping and sharpening can be performed onpixel luminance (Y) values and then applied to the median filtered andcolor corrected pixel color component values. Since the luminance (Y)values are preserved in the median filtering calculations, minimal sideeffects result from performing the tone mapping and sharpening inparallel with or during time periods that at least partially overlapmedian filtering operations on the same pixels.

The pixel neighborhoods used for median filtering can be in 3×3 or 5×5neighborhoods that are centered on the pixel being filtered andfiltering is preferably applied to pixels in the image that are includedin the final image and for which the specified neighborhood of pixelsover which the median filtering is performed is fully populated. Thismay exclude pixels at the border of the image that may be used withmodified median filtering or without median filtering.

The logarithms of pixel color component values and of pixel luminancevalues are preferably calculated before dividing by luminance so thatthe log of the ratio may be calculated as the difference oflogarithmically encoded color component and luminance values toeliminate the need to perform division operations.

The above can be implemented as a modification of embodiments alsodescribed herein. In one embodiment, the logarithm of the ratio of pixelred or blue color components to their respective green color componentwas used in place of the logarithm of the ratio of the pixel red or bluecolor component to their respective pixel luminance value as used inthis implementation. It can be beneficial to clamp the ratio values athigh and low threshold limits and to use the median calculation thatreuses comparison results and partial comparison tallies for finding themedians for successive pixels. The median filtering operations can beperformed using the same calculations and organization indicated for thegreen pixel referenced calculation by substituting the pixel luminancevalue with the green pixel value in the median filtering operation. Themedian filtering is still done separately for separate color planes.Optionally the color space may be transformed to other forms such asYCbCr with the Y used as luminance and the median filtering applied tolog(Cr/Y) and separately to log(Cb/Y).

In high dynamic range images, pixel values may vary by more than 1000 toone even in 3×3 or 5×5 arrays of pixel values used to calculateinterpolated pixel components. A portion of the interpolated values mayprovide reasonable representations of actual colors in the scene; but,with the large ratios of closely spaced pixels that are present in manyhigh dynamic range images, some interpolated values deviate greatly fromactual color component values in the scene. High dynamic range imagestend to be marred by a sprinkling of pixels having interpolation errorsthat really stand out visually in the interpolated image. One exemplarybenefit of the median filter is to replace a majority of themis-interpolated pixel color component values that really stand out withvalues that fit visually in the image and which are in fact likely to betruer to the actual image. Two of three color components are normallysupplied by interpolation at each pixel site and in many cases wherethere is a serious error in interpolation, one of the interpolated colorcomponents is much more seriously in error than the other. Thus, therecan be benefit in performing the median filtering using color planes ofactual interpolated values rather than of color values from a colorspace that is transformed from the one in which the image was acquired.By using color components from the color space in which the image isacquired, the color component that is seriously in error tends to bereplaced directly rather than indirectly after first blending the errorwith multiple color components in a color space transformation. Thissaid, the pixel values can be normalized to a luminance reference sothat the median filtering is performed primarily on the color plane'schroma component while preserving the original luminance value for thepixel.

Typical prior art median filtering has been performed by subtracting aluminance component from a color value that contains combined chroma andluminance information. The use of subtraction without intentionallyconverting the values to logarithmic form misses the mark in processinghigh dynamic range images where the luminance component of pixels in theneighborhood over which the median is calculated may vary greatly. Thisis especially evident since the greatest benefit from median filteringis usually associated with borders of bright lights in a scene or aroundspecular reflections from the sun in a daytime scene or at a boundarysuch as a window frame between regions where the ratio of brightnesslevels may be extreme. In these cases, contrast ratios even in smallneighborhoods over which median values are selected may be very largeand these are also the areas in the image that normally contain anexcessive number of pixels that have interpolation errors that reallystand out. Use of the ratio of the pixel value to luminance or to aluminance related value (such as green) is much more satisfactory and inmost of the instances that were investigated, the use of luminance as areference yielded better results than the use of green for the medianpixel reference value. By converting the pixel and luminance relatedpixel related values to logarithmic form, a subtraction becomesequivalent to a division or ratio calculated in linear space so that thesubtraction may be used in this distinct case to provide the equivalentof a division or a ratio in linear space. Since the logarithm of a valueincreases monotonically with increases in the value, the ranking and,thus, the selection provided by the median filter remains substantiallyunchanged when logarithmically encoded pixel related values aresubjected to median filtering in place of linearly encoded pixel relatedratios. Thus, the use of logarithmically encoded values can serveprimarily to permit substitution of subtraction for division to obtainan indication of the ratio without substantially changing the result ofthe filtering.

In the alternate embodiment, green pixel values were used as a pseudoluminance reference. In one embodiment, luminance values replace thegreen pixel values in the ratios and after the median filteringoperation the green pixel value is preferably calculated from theoriginal pixel luminance and the median filtered red and blue filteredpixel values so that the pixel luminance rather than the pixel greenvalue remains unchanged by the median filtering operation.

The effectiveness of the filtering in replacement of errantlyinterpolated pixel color component values with ones that fit the imageis generally improved by using pixel luminance values rather than greenpixel component values in the median filtering operation. A secondexemplary advantage is that the same logarithm of the luminance is usedfor a tone mapping operation that can be performed at the same time asthe median filtering. The same logarithmically encoded luminance valuesmay be used for both the tone mapping and the median filteringoperations and since the pixel luminance values remain constant in theluminance referenced median filtering operation, the errors introducedby having the luminance values altered in the median filter so that theyno longer match those used in the tone mapping are eliminated.

FIG. 7 is a diagram that depicts the relationship of the luminancereferenced median color filter in accordance with one embodiment toother high dynamic range image processing operations that are performedon the image. Pixel values from high dynamic range camera 701, afteroptional processing for color balance and color temperature areinterpolated to provide missing color components 702 and the logarithmsof the pixel luminance values are calculated 703. The logarithms ofluminance values are provided for tone mapping 705 and for the medianbased color filter 706 which also uses red and blue color componentsthat are provided from the color interpolation block 702. Following tonemapping 705, the tone mapped pixel luminance values can be clamped 707so that they are non-negative and so that they are further restricted tothe numerical working range that is provided. The working rangepreferably includes the luminance values that are not likely to resultin saturation of the pixel in further processing stages. The clamped,tone mapped values can be sharpened 711 and again clamped 713 to anon-negative working range that can include the pixel values that arenot likely to saturate in further pixel processing stages. In aprocessing path that can be executed while tone mapping and sharpeningoperations are executed for the same rows of pixels, the logarithm ofthe luminance 703 and corresponding pixel color components 704 arefiltered by median filter 706 and the median filtered color componentscan be expressed in logarithmic form as the ratios of the medianfiltered values R′ and B′ to the pixel luminance log₂(R′/Y) and oflog_(e)(B′/Y) are buffered 708 to synchronize their use with colorcorrection and additionally with completion of tone mapping andsharpening that is performed on luminance values for the correspondingpixels. Just prior to color preservation and recombination with the tonemapped and sharpened pixel luminance value Y_(C)″ in blocks 715, 716,and 717, the antilogarithms of log₂(R′/Y) and of (log₂(B′/Y) can becalculated to obtain R′/Y and B′/Y (710) and G′/Y can be derived fromthem (712). G′/Y can be clamped so that it is non-negative and the rangemay be further restricted to represent a practical range for the G′/Ycolor component value. This intermediate clamping operation before colorcorrection may be of benefit and may justify calculation and clamping ofthe green color component before performing color correction so somepixel saturation is still to be expected and often distorts colors in orbleaches portions of the directly viewed lights in the scene. Colorcorrection 714 is performed to obtain color corrected color to luminanceratios R″/Y, G″/Y, and B″/Y to obtain color corrected pixel colorcomponent values. Alternatively, the derivation of G′/Y (712) may becombined with the color correction calculation 714. As part of colorcorrection, the values can be clamped to positive, non-zero values andconverted to a logarithmic form.

With a camera 701 having a very high dynamic range, color data iscaptured for most lights even when they are in direct view. Even thoughtone mapping normally preserves most of the dynamic range of thecaptured image, it is based on locally averaged luminance and does notinclude the effects of sharpening, color correction, and the fact thatindividual pixel color components expressed in the RGB color space oftenhave pixel color component values that exceed the luminance values.Color is preserved by obtaining the value of each pixel's maximum colorcomponent and taking its ratio to the saturation value (e.g., themaximum display level) for the pixel color component. If the ratio isgreater than one indicating that it exceeds the saturation level,substantially all of the color components of the pixel can be divided bythe ratio so that the maximum color component in the scaled pixel valuejust matches the output saturation level resulting in proper display ofthe color component. This can be especially beneficial for automotivevisual display or for color based processing of the images forautomotive use in order to preserve the color of lights and bright signsand markings that are given special safety related interpretations. Thevisual appeal of the image is also enhanced.

The numerous multiply and divide operations that are used in the colorpreservation may be more efficiently performed by converting the pixelrelated luminance and color component values to logarithmic form so thatmultiplies and divides are replaced by adds and subtracts, respectively,and comparison orders are preserved with the ratio of one in linearspace represented by a logarithmic value of zero. Thus, the logarithm ofluminance is provided as the sharpened pixel luminance output valuelog₂(Y″) (Block 713) and the logarithmically encoded color correctedcolor component values expressed as ratios to pixel luminancelog₂(R″/Y), log₂(G″/Y), and log₂(B″/Y) are provided by block 714 aftermedian filtering, calculation of G′/Y and color correction. In otherembodiments, the tone mapping operation output was in the form of pixelluminance modification factors, Y′/Y and the original pixel RGB colorvalues can be multiplied by this factor to obtain the tone mapped pixelcolor component values. With Y available in the tone mapping andunchanged by the luminance ratio based color median filter, an output ofthe tone mapping operation for the embodiment of FIG. 7 can be toprovide the tone mapped luminance value Y′ in place of the ratio Y′/Y.Y′ may be directly input to the sharpening filter without the necessityto obtain tone mapped values of R, G, and B and then calculate a newluminance value from them. Secondly, the modified median filter operateson the logarithm of the ratio of a selected pixel color component to itsluminance and serves to find the median value of the logarithmicallyencoded color to luminance ratios in the specified neighborhood of thetarget pixel and to replace the logarithmically encoded color componentto luminance ratio of the pixel color component being filtered with themedian value to provide the essential step of the color median filteringoperation. As with another embodiment, the logarithm of the color ratiocan be clamped between positive and negative limits that need not bevalues of equal magnitude to represent a practical range for the colorto luminance ratios. The limits can be modified since the luminance canbe used in place of green for the denominator term of the ratio. Theclamped, logarithmically encoded median filtered red and blue colorratios to original pixel luminance after clamping are the values thatcan be buffered. For example, the value used for Y in the examples iscalculated using Y=0.299R+0.587G+0.114B so the ratios should never behigher than about 8.8 for blue, 3.3 for red and 1.7 for green. Since fewcolors in nature or in safety related indicators are pure blue, it maybe reasonable to limit the integral portion to three bits to includevalues just under 8 and to include more of the range for fractionalvalues and to add accuracy. The limiting needs to be repeated more oftenif done before finding median values but this has the benefit ofrestricting the bit width of compare circuits used to find the medianvalues.

The systems and methods described herein can be implemented using anon-transitory computer readable medium having stored thereon softwareinstruction that, when executed by a processor, cause the processor todisplay a captured image on a display having a lower dynamic range thanthe imaging system by executing one or more steps.

Examples of high dynamic range imagers that can be used are described inUnited States Patent Application Publication Numbers US 2008/0192132 A1,US 2009/0256938 A1, US 2009/0160987 A1, US 2009/0190015 A1, US2010/0187407 A1, and US 2010/0188540 A1, all entitled “IMAGING DEVICE,”the entire disclosures hereby being incorporated herein by reference.

Modifications of the invention will occur to those skilled in the artand to those who make or use the invention. Therefore, it is understoodthat the embodiments shown in the drawings and described above aremerely for illustrative purposes and not intended to limit the scope ofthe invention, which is defined by the following claims as interpretedaccording to the principles of patent law, including the doctrine ofequivalents.

What is claimed is:
 1. An imaging system configured to display acaptured image on a display having a lower dynamic range than theimaging system, the imaging system comprising: a high dynamic rangeimager configured to capture at least one high dynamic range image; anda processing device communicatively connected to said high dynamic rangeimager, wherein said processing device is configured to rank eachelement of an array with respect to an ordering criterion and select amedian element, where the median element is a function of a ratio of anon-reference color component of a pixel to a reference color componentof the pixel, wherein said ordering criterion comprises a discreteranking value assigned to every element.
 2. The imaging system of claim1, wherein said discrete ranking of equal pixel values is based upon atleast one of an order of entry into said array and a position withinsaid array of compare values.
 3. The imaging system of claim 1, whereinsaid processing device comprises a field programmable gate array.
 4. Theimaging system of claim 1, wherein the display of said imaging systemreplaces an interior rearview mirror in a vehicle.
 5. The imaging systemof claim 1, wherein said processing device is configured to filtersuccessive pixels so pixel related values introduced into a neighborhoodof pixels over which a median or value of designated rank is chosen areused with related comparison results.
 6. The imaging system of claim 5,wherein said processing device is further configured such thatcomparisons are organized so only pixel related elements in subsets arenewly introduced to an array and are compared against other pixelrelated elements in said array.
 7. The imaging system of claim 5,wherein said processing device is further configured to compare resultsobtained once and used in finding said median or said value ofdesignated rank for more than one pixel in an array.
 8. An imagingsystem configured to display a captured image on a display having alower dynamic range than the imaging system, the imaging systemcomprising: a high dynamic range imager configured to capture at leastone high dynamic range image; and a processing device communicativelyconnected to said high dynamic range imager, wherein said processingdevice is configured such that elements with equal values are sorted ina prescribed sequence so each element has a rank that is unique within aset, wherein a middle element is always specified.
 9. The imaging systemof claim 8, wherein said processing device comprises a fieldprogrammable gate array.
 10. The imaging system of claim 8, wherein thedisplay of said imaging system replaces an interior rearview mirror in avehicle.
 11. The imaging system of claim 8, wherein said processingdevice is configured to rank each element of an array with respect to anordering criterion and select a median element, where the median elementis a function of a ratio of a non-reference color component of a pixelto a reference color component of the pixel, wherein said orderingcriterion comprises a discrete ranking value assigned to every element.12. The imaging system of claim 11, wherein said discrete ranking ofequal pixel values is based upon at least one of an order of entry intosaid array and a position within said array of compare values.
 13. Theimaging system of claim 8, wherein said processing device is configuredto find a median from a set of pixel related values and provide anenhanced pixel metric based at least partially upon modified ratios ofpixel color components to at least one of pixel color component andluminance values.
 14. The imaging system of claim 8, wherein saidprocessing device is configured to filter successive pixels so pixelrelated values introduced into a neighborhood of pixels over which amedian or value of designated rank is chosen are used with relatedcomparison results.
 15. An imaging system configured to display acaptured image on a display having a lower dynamic range than theimaging system, the imaging system comprising: a high dynamic rangeimager configured to capture at least one high dynamic range image; anda processing device communicatively connected to said high dynamic rangeimager, wherein said processing device is configured to use a secondderivative term calculated from one of a red pixel and a blue pixel forinterpolation of a green pixel.
 16. The imaging system of claim 15,wherein said processing device is configured such that elements withequal values are sorted in a prescribed sequence so each element has arank that is unique within a set, wherein a middle element is alwaysspecified.
 17. The imaging system of claim 15, wherein said processingdevice is configured to rank each element of an array with respect to anordering criterion and select a median element, where the median elementis a function of a ratio of a non-reference color component of a pixelto a reference color component of the pixel, wherein said orderingcriterion comprises a discrete ranking value assigned to every element.18. The imaging system of claim 15, wherein said discrete ranking ofequal pixel values is based upon at least one of an order of entry intosaid array and a position within said array of compare values.
 19. Theimaging system of claim 15, wherein said processing device comprises afield programmable gate array.
 20. The imaging system of claim 15,wherein the display of said imaging system replaces an interior rearviewmirror in a vehicle.